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GENERALIZED AND DEGENERATE WHITTAKER 

MODELS 


RAUL GOMEZ, DMITRY GOUREVITCH, AND SIDDHARTHA SAHI 

Abstract. We study generalized and degenerate Whittaker models for 
reductive groups over local fields of characteristic zero (archimedean or 
non-archimedean). Our main result is the construction of epimorphisms 
from the generalized Whittaker model corresponding to a nilpotent or¬ 
bit to any degenerate Whittaker model corresponding to the same orbit, 
and to certain degenerate Whittaker models corresponding to bigger or¬ 
bits. We also give choice-free definitions of generalized and degenerate 
Whittaker models. Finally, we explain how our methods imply analo¬ 
gous results for Whittaker-Fourier coefficients of automorphic represen¬ 
tations. 

For GL„(F) this implies that a smooth admissible representation tt 
has a generalized Whittaker model Wc>(7r) corresponding to a nilpotent 
coadjoint orbit O if and only if O lies in the (closure of) the wave- 
front set WF(7r). Previously this was only known to hold for F non- 
archimedean and O maximal in WF(7r), see |MW87| . We also express 
VVc>(7r) as an iteration of a version of the Bernstein-Zelevinsky deriva¬ 
tives |BZ77irAGS15a| . This enables us to extend to GL„(R) and GL„(C) 
several further results from |MW87| on the dimension of VVc>(7r) and on 
the exactness of the generalized Whittaker functor. 
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1. Introduction and main results 

1.1. General results. Let IF be a local field of characteristic zero, G a 
reductive group defined over F, g its Lie algebra and g* the dual space 
to g. A Whittaker pair is an ordered pair {S, ip) G g x g* such that S is 
semi-simple with eigenvalues of the adjoint action ad{S) lying in Q, and 
ad*{S){(p) = —2ip. Note that ip is necessarily nilpotent and given by the 
Killing form pairing with a (unique) nilpotent element / = /(^ G g- Following 
|MW87] we attach to {S, ip) a certain smooth representation of G called 
a degenerate Whittaker model for G. 

Two classes of Whittaker pairs and the corresponding models will be 
of special interest to us. If S' is a neutral element for (see Dehnition 
12.2.21 below), then we will say that (S', (/?) is a neutral pair and call a 

neutral model or a generalized model (see |Kaw85l IMW871 IYa86[ IGZ14] L 
The second class consists of Whittaker pairs (S, ip) where S is the neutral 
element of a principal s[ 2 -triple in G; in this case is necessarily a principal 
nilpotent element for a Levi subgroup of G, and we will say (S, ip) is a 
PL pair, and is a PL model or a principal degenerate model (see 

We will now sketch the definition of 'Ws,(p^ referring to [12.5l below for more 
details. Let u C g denote the sum of all eigenspaces of ad{S) with eigenvalues 
at least 1. Note that u is a nilpotent subalgebra and let U := Exp(u) C G 
be the corresponding nilpotent subgroup. Fix an additive character of F. 
Suppose first that 1 is not an eigenvalue of ad{S). Then the restriction of ip 
to u is a character of u, which defines a character x<p of U. The degenerate 
Whittaker model is defined to be the Schwartz induction of this character: 
kVs,v 3 := If 1 is an eigenvalue of ad{S) then consider the anti¬ 

symmetric form on u given by ujp{X,Y) := ip{\X,Y]) and let n denote the 
radical of this form. Let nG= n H Ker(/j, and let N' := Exp(n^). It is easy 
to show that N' is a normal subgroup of U and U/N' is isomorphic to a 
(generalized) Heisenberg group, of which ip defines a central character Xp- 
Let ap denote the oscillator representation of U/N' with central character 
Xp- Gonsider as a representation of U and define Ws^p ■= ind^ ap. 
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If {S, ip) is a neutral pair then the generalized Whittaker model yVs,ip does 
not depend on the choice of a neutral S, and will thus be denoted Since 
conjugate nilpotent elements give rise to isomorphic generalized Whittaker 
models, we will also use the notation Wo for a nilpotent coadjoint orbit O. 

We denote by Gs^ C Q* the closure of the orbit of (p under the coadjoint 
action of the centralizer of S in G. 

Theorem A (See Let {S,(p) be a Whittaker pair and let ip G Gsp- 
Then there is a G-equivariant epimorphism ofW^p onto Ws,p- 

In particular, taking ip = ip we see that the generalized Whittaker model 
maps onto any degenerate Whittaker model corresponding to the same ip. 
In fact, we prove a more general result (Theorem I.S.O.ll) on epimorphisms 
between pairs of degenerate Whittaker models, which enables one to define 
a preorder on the pairs (S', p). If we fix ftp to be a regular nilpotent element 
for a Levi subgroup of G then the minimal elements under this preorder are 
the PL Whittaker pairs (see 113.3p . The corresponding principal degenerate 
Whittaker models are inductions of (possibly degenerate) characters of the 
nilradicals of minimal parabolic subgroups. 

The above results have applications to the study of Whittaker functionals 
on representations of G. Following |GZ14j we briefly recall the necessary 
background. 

Let M.{G) denote the category of smooth admissible^ (finitely generated) 
representations of G (see |BZ76l ICas89[ IWall92| l. For tt G M.{G) and a 
nilpotent orbit O C g denote 

(1) Wo(7r) := HomG(Wo,7r*). 

The study of Whittaker and generalized Whittaker models for represen¬ 
tations of reductive groups over local fields evolved in connection with the 
theory of automorphic forms (via their Fourier coefficients), and has found 
important applications in both areas. See for example |Sh74[ INPS731 iKosTSl 
IKaw85[ IYa86[ IWall88al IGinObl I.Iian7[ IGRSIlj . From the point of view of 
representation theory, the space of generalized Whittaker models may be 
viewed as one kind of nilpotent invariant associated to smooth representa¬ 
tions. Another important invariant is the wave front cycle: 

(2) WFC(7r) = E co(»)Pi. 

ocr 

nilpotent 

dehned by Harish-Chandra in the non-archimedean case and by Howe and 
Barbasch-Vogan in the archimedean case f |How81l IBVSOj : see also |Ros95[ 
ISVOOj l. Recently, the behaviour of the wave-front set and the generalized 
Whittaker models under ^-correspondence was studied in |GZ14l iLoMalS] . 

For IF non-archimedean, Moeglin and Waldspurger |MW87] have estab¬ 
lished that WFC('7r) completely controls the spaces of generalized Whittaker 
models of interest, namely, the set of maximal orbits in WFC(7r) coincides 


^If F is archimedean then by admissible we mean admissible Frechet representation of 
moderate growth. 
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with the set of maximal orbits such that Wc:)(t) 7 ^ 0 , and for any orbit O 
in this set we have 

cc>(7J') = dim>Vc)( 7 r). 

In |MW87| it is assnmed that the residne characteristic is odd. This as¬ 
sumption was recently removed in |Varl4j . In [Moe96| . the main result of 
[MW87j is used in order to prove that for classical gronps, the maximal or¬ 
bits in the wave-front set of a tempered representation are distinguished, and 
the maximal orbits in the wave-front set of any admissible representation 
are special. The latter was recently generalized in |.ILSj . Partial analogs 
of these results hold also for archimedean F. see [Har m for the former and 
[BV821IBV8311.IosSO] for the latter. 

For archimedean F, the correspondence between the wave-front set and 
non-vanishing of degenerate Whittaker models is not yet (fully) understood, 
except for several special cases inclnding the representations with the largest 
Gelfand-Kirillov dimension |Vog78[ IMat92| and unitary highest weight mod¬ 
ules [YaOlj . For the latter, the wave front set was computed earlier in 
|Pr91aj . In |Mat87j . it is shown in fnll generality that every orbit O with 
VV’c)( 7 r) 7 ^ 0 lies in the Zariski closure of some orbit in WFC( 7 r). The paper 
|GSI5j proves an expected existence of non-zero maps from principal degen¬ 
erate Whittaker models to admissible representations, and by Proposition 
13.3.41 below, any other degenerate Whittaker model corresponding to the 
same orbit is mapped onto a principal degenerate Whittaker model. Let 
WF( 7 r) denote the closnre in the local field topology of the nnion of all 
the orbits in WFG( 7 r). In 113.31 we review the results of |Mat87[ iGSlSj and 
deduce, using Theorem]^ the following theorem. 

Theorem B f 113.31) . Let G he a complex reductive group and let vr G A4(G). 
Let (S, ip) be a Whittaker pair such that ip is given by Killing form pairing 
with a prineipal nilpotent element of the Lie algebra of a Levi subgroup of 
G. Then 

Ws,(^(vr) 7 ^ 0 G WF(7r). 

If G is classical then the set WF (vr) is nniquely determined by its intersec¬ 
tion with the set of “principal Levi” nilpotents ip, see [GSI51 Theorem D]. A 
result related to Theorem iBl is proved by Matumoto in [Mat9nj . Namely, let 
G be a complex rednctive gronp and let vr G A4(G) have regnlar infinitesimal 
character. Let (S', ip) be a Whittaker pair and assnme that the orbit of ip 
is dense in WF( 7 r), and also that it contains a dense subset of the nilradi- 
cal of the parabolic subgroup defined by S. Then 0 < dimW 5 ,(^( 7 r) < 00 . 
Matumoto also proves the vanishing of the corresponding higher homology 
groups. 

For F = M a weaker version of TheoremlBlholds. see Corollary 13. 3. 7l below. 

In Section Owe give choice-free definitions of degenerate Whittaker mod¬ 
els. These definitions use the Deligne filtration instead of the Jacobson- 
Morozov theorem and thus might be suitable for local fields of positive 
characteristic. 

In the global case, instead of degenerate Whittaker models one considers 
explicit functionals on automorphic representations defined by integration 
against a character of a nilpotent subgroup. Such functionals are called 
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Whittaker-Fourier coefficients and denoted In Section [ 6 ] we give 

the definitions and explain how to adapt our arguments to the global case 
and deduce the following theorem. 

Theorem C (See Let K he a number field, let G be the group of adelic 
points of a reductive group defined over K and g be its Lie algebra. Let tt he 
an automorphic representation ofG. 

Let G 0 X g* be a Whittaker pair. Suppose that WFs,^p{f) 0 for 

some / G TT. Then WF^{f') 0 for some /' G vr. 

1.2. Further results for GL„(F). For Gn := GL„(F) we show that The¬ 
orem allows one to compare degenerate Whittaker models corresponding 
to any nilpotent orbits O, O' C g* := 0 [(n,F)* s.t. O C O'. To that end we 
prove the following geometric theorem. 

Theorem D l li4.2p . Let 0,0' C g* be nilpotent coadjoint orbits with 
O C O'. Then there exists a Whittaker pair {S,(p) such that a O' and 
O intersects Gsfi- Moreover, S can he chosen to he diagonal with integer 
eigenvalues and (p can be chosen to he given by the trace pairing with a 
matrix in Jordan form. 

Remark 1.2.1. An analogous statement holds for some pairs of nilpotent 
orbits O <Z O' in other classical groups (see Remark 14.3.21 belowl. but not 
in general. Indeed, it can be shown that if Ggip intersects a distinguished 
orbit O then 0 = 0'. 

Using Theorem El [M at87] . and Gorollarv 13.8.71 for archimedean F, and 
Theorems El and El together with the results of [MW871 IVarl4] for non- 
archimedean F we deduce the following theorem. 

Theorem E 1 114.81) . Let TT G Ad{Gn). Let O G be a nilpotent orbit. Then 
we have 

(3) Wo(7r) / 0 44 O C WF(7r). 

Remark 1.2.2. It would be interesting to know to what extent this theo¬ 
rem holds for other reductive groups. For archimedean F, Theorem El and 
Gorollary 13.3.71 provide a partial extension. In the non-archimedean case, 
the failure of Theorem El for general groups, as noted in Remark 11.2.11 
represents the main obstacle for extending our approach. 

In H4.4l we give a precise description of the generalized Whittaker spaces in 
terms of certain functors introduced in [AGSlhal IAGS15b] in connection 
with the generalization of the theory of Bernstein-Zelevinsky derivatives to 
the archimedean setting. We will also use related functors that go in the 
other direction. We refer to 114.41 below for the precise definitions of both 
functors. 

Theorem F 1 114.41) . Let A = (Ai,..., A^) be a partition of n and 0\ C g* 
be the corresponding nilpotent orbit. Then 

(4) {£)■■■), 

where C denotes the one-dimensional representation of the trivial group Gq, 
and for vr G M.{Gn) we have 

(5) Wo,( tt) ~ (F^''(---£;^i(7r) •••))*• 
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Let O C 0 * be a nilpotent orbit. Let Ai-^{Gn) denote the Serre subcat¬ 
egory of A4(Gn) consisting of representations with wave-front set inside the 
closure of O and let M^‘^{Gn) denote the Serre subcategory of M-^{Gn) 
consisting of representations with wave-front set not containing O. Let 
■M.^{Gn) '■= (Gn)(Gn) denote the quotient category. 

Using the main results of |AGS15al IAGS15bl IGS131 IGS15| we obtain 

Corollary G l Ud.hp . fij The functor n WoiT^) defines an exact faith¬ 
ful functor from A4^{Gn) to the category of finite dimensional vector 
spaces. 

(a) // TT G A4-^{Gn) is an irreducible unitarizable representation, or a 
monomial representation, and tt ^ A4'^^(Gn) then Wc)( 7 r) is one¬ 
dimensional. 

This corollary is new only in the archimedean case, since over p-adic fields 
exactness is well-known and hniteness of dimension is shown in |MW87j . It is 
also shown in |MW87] that WoiT^) is one-dimensional for all irreducible vr G 
A4^{Gn). Over archimedean fields, Wo( 7 r) is clearly not one-dimensional 
for any irreducible tt of finite dimension bigger than one. 

Let us also comment that for any irreducible vr G A4{Gn), WF( 7 r) is the 
closure of a single nilpotent orbit. Over archimedean fields this follows from 
|BB82[ l,Tos85| and over non-archimedean fields this is proven in [MW87| . 
More generally, it follows from [.Tos85j that for any real reductive group G 
and any irreducible tt G Ai{G), all the maximal orbits in WF( 7 r) lie in a 
single complex nilpotent orbit. An analogous statement is conjectured over 
p-adic fields but not proven yet. 

We conjecture that the functor vr i-A Ydoij^) is exact for all reductive 
groups, and we hope to prove this in the future, generalizing the technique 
of |AGS15bj . 

1.3. The structure of our proofs. Let us first describe the idea of the 
proof of Theorem lAl in the case (p = (p. We first show that S can be presented 
as h-\- Z, where h is a neutral element for (p and Z commutes with h and (p. 
Then we consider a deformation St = h -\-tZ, and denote by Ut the sum of 
eigenspaces of ad{St) with eigenvalues at least 1 . We call a rational number 
0 < t < 1 regular if = Ut+e for any small enough rational e, and critical 
otherwise. Note that there are finitely many critical numbers, and denote 
them by ti < • • • < tn- Denote also to := 0 and tn+i '■= 1. For each t 
we define two subalgebras lt,Xt C Ut. Both h and are maximal isotropic 
subspaces with respect to the form Vt contains all the eigenspaces of 
Z in Ut with positive eigenvalues and h contains all the eigenspaces with 
negative eigenvalues. Note that the restrictions of (p to h and tt define 
characters of these subalgebras. Let Lt := Exp([t) and Rt := Exp(rt) denote 
the corresponding subgroups and X(r denote their characters defined by (p. 
The Stone-von-Neumann theorem implies 

Wst,^ - indf^(x^) ^ ind^ix^)- 

We show that for any 0 < i < n, tt. C \ti+x ■ This gives a natural epimor- 
phism 
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Altogether, we get 

If (fi (fl, we identify g ~ g* using a non-degenerate invariant form and 
complete (p to an s/ 2 -triple (e, h, p) such that h commutes with S. Then we 
show, using the Slodowy slice, that the conditions imply that (p is conjugate 
under Gs to p+p' with ad*{e){p') = 0. We finish the proof by a deformation 
argument similar to the case p = p. 

For Theorem iBl the implication W 5 , 1 ^( 71 ) 7 ^ 0 => y? G WF( 7 r) follows from 
|Mat87| . To prove the other direction, we show that y^s,tp maps onto some 
principal degenerate Whittaker model kV^ Thus the theorem follows from 
the nonvanishing of W^j^( 7 r) (under the condition p G WF( 7 r)), which was 
shown in |GS15j . The same argument gives Corollary 13.3.71 - an analogous 
statement for quasisplit real reductive groups. 

For the proof of Theorem [D] we identify g* with g^ using the trace form, 
and parameterize nilpotent orbits by partitions. Then we prove the theorem 
for partitions of length two by an elementary matrix conjugation argument. 
We hnish the proof by induction. The induction argument, however, is not 
so easy since the statement is not “transitive”. For any pair of partitions 
A < /i (where < refers to the natural order on partitions which corresponds 
to the closure order on orbits), we consider two pairs of partitions of two 
smaller numbers that add up to a number bigger than n. Then we take 
S' and S" corresponding to the two pairs of partitions and force them to 
coincide on the joint block by adding a scalar matrix to one of them. In this 
way we obtain a diagonal matrix S G gln(^) that satisfies the requirements 
of the theorem. 

For archimedean F, Theorem E follows from Theorem iBl [Mat87] . and 
Corollary 13.3.71 For non-archimedean F, it was shown in [MW87] that 
7 ^ 0 for any Whittaker pair {S,p) such that the orbit of is a 
maximal orbit in WFC(7r). If O C WF(7r) then O C O' for some maximal 
orbit O' G WFC(7r). Pick {S,p) that correspond to 0,0' by Theorem [Dj 
Then, by Theorem lAl VV’ 5 ,,^( 7 r) embeds into WoiTr) and thus Wc)( 7 r) 7 ^ 0. 

We prove Theorem E by induction on k. We let X' := (A 2 ,..., Afc), and 
by the induction hypothesis get 

Wo,, 

Thus, in order to prove dU we have to show that 
( 6 ) Wo, 

We note that both sides of the formula are isomorphic to inductions of the 
same character from two nilpotent subgroups that differ only in the last 
Ai columns. Then we prove ® by a deformation argument similar to the 
proof of Theorem lAl Finally, ([S]) follows from ([1]) by a version of Frobenius 
reciprocity. 

Corollary O follows from ([5|) using the properties of archimedean pred¬ 
erivatives proven in [ACSlSal lACSlSbl ICS131 ICSlSj . 

The proof of Theorem O is analogous to the proofs of Theorems |Dj 
The only difference is that we cannot apply the Stone-von-Neumann theorem 
since in the global case we consider Whittaker-Fourier coefficients, that are 
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some explicit functionals on an automorphic representation. We replace it 
by Lemma 16.0.21 that is proven by an explicit integral transform followed 
by a Fourier transform on a compact abelian group. This lemma is in the 
spirit of [GRSlll Propositions 7.2 and 7.3]. 

If F is archimedean, one can consider more general models, and analogs 
of Theorems EllFl will remain valid for them, see Remark 12.5.41 
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2. Preliminaries 

2.1. Notation. For a semi-simple element S and a rational number r we 

denote by the r-eigenspace of the adjoint action of S and by g>j. the sum 
0r'>r Sr" notation (g*)^ and (g*)f^ for the correspond¬ 

ing grading and hltration of the dual Lie algebra g*. For A G g or A G g* 
we denote by gjv the centralizer of A in g, and by Gx the centralizer of A 
in G. We say that an element h G g is rational semi-simple if its adjoint 
action on g is diagonalizable with eigenvalues in Q. 

If (/,/i,e) is an 5[2-triple, we will say that e is a nil-positive element for 
h, / is a nil-negative element for h, and h is a neutral element for e. For 
a representation V of (/, h, e) we denote by P® the space spanned by the 
highest-weight vectors and by the space spanned by the lowest-weight 
vectors. 

From now on we fix a non-trivial unitary additive character 
(7) X : F ^ 

such that if F is archimedean we have x{x) = exp(27ri37(x)) and if F is 
non-archimedean the kernel of x is the ring of integers. 

2.2. s[ 2 -triples. We will need the following lemma which summarizes sev¬ 
eral well-known facts about s[ 2 -triples. 


Lemma 2.2.1 (See |Bou75l §11] or |Kos59j L 
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(i) Any nilpotent element is the nil-positive element of some 5l2-triple in 

0 - 

(a) If h has a nil-positive element then e is a nil-positive element for h if 
and only */e € 02 ad{e) defines a surjection 0 o-» 02 - The set of 
nil-positive elements for h is open in 02 - 
(Hi) If e is nilpotent then h is a neutral element for e if and only i/e € 02 
and h € Im(a(i(e)). All such h are conjugate under G^- 
(iv) If (/, h, e) and (/', h, e) are 5l2-triples then f = f. 

('^) If is an 5\2-triple and Z commutes with two of its elements 

then it commutes also with the third one. 

It is easy to see that the lemma continues to hold true if we replace the 
nil-positive elements by nil-negative ones (and 02 by 0 (^ 2 )- 

Definition 2 . 2 . 2 . We will say that h G 0 is a neutral element for ip ^ q* if h 
has a nil-positive element in 0 , G ( 0 *)- 2 ) linear map 0 q —)• {q *)^2 

given by x I—)■ ad*{x){p) is an epimorphism. Note that if we identify 0 with 
0 * (in a G-equivariant way) this property becomes equivalent to p being a 
nilnegative element for h, or —h being a neutral element for p. We also say 
that 0 G 0 is a neutral element for 0 G 0 *. 

2.3. Schwartz induction. 

Definition 2 . 3 . 1 . If G is an /-group, we denote by Rep°°(G) the category 
of smooth representations of G. If HI C G is a closed subgroup, and vr G 
Rep°°(fl) is a smooth representation of G we denote by ind^( 7 r) the smooth 
compactly-supported induction as in [BZ761 §2.22]. 

If G is an affine real algebraic group, we denote by Rep°°(G) the category 
of smooth nuclear Frechet representations of G of moderate growth. This 
is essentially the same definition as in |dC191[ §1.4] with the additional 
assumption that the representation spaces are nuclear (see e.g. [Tre67[ §50]). 
If 77 C G is a Zariski closed subgroup, and vr G Rep°°(77) we denote by 
ind^( 7 r) the Schwartz induction as in |dC191l § 2 ]. More precisely, in [dC191] 
du Cloux defines a map from the space 5(G, vr) of Schwartz functions from 
G to the underlying space of tt to the space G°°(G, vr) of all smooth vr-valued 
functions on G by / e-)• / where 

J{x) = [ TT{h)f{xh)dh, 

Jh&H 

and dh denotes a fixed left-invariant measure on H. The Schwartz induction 
ind^( 7 r) is defined to be the image of this map. 

From now till the end of the subsection let G be either an /-group or an 
affine real algebraic group, and 77' C 77 C G be (Zariski) closed subgroups. 

Lemma 2 . 3.2 f |BZ76[ Proposition 2.25(b)] and |dC191l Lemma 2.1.6]). For 
any vr G Rep°°(i7') we have 

ind^/(7r) ~ ind^ ind|^/(7r). 

Corollary 2 . 3 . 3 . For any vr G Rep°°(77) we have a natural epimorphism 
ind^/ {tt\h')^ (vr). 
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Lemma 2.3.4. Let p G Rep°°(-?/), tt G Rep°°(G) and let vr* denote the dual 
representation, (endowed with the strong dual topology in the archimedean 
case). Then 

HomG(indg(/9),7r*) = Hom/^(p, 
where Ah and Aq denote the modular functions of H and G. 

The non-archimedean case of this lemma follows from |BZ76[ Proposition 
2.29]. We prove the archimedean case in Appendix lAl We will only use this 
lemma in the case when G is reductive, vr G Ai{G), H is nilpotent and p is 
one-dimensional. 

2.4. Oscillator representations of the Heisenberg group. 

Definition 2.4.1. Let Wn denote the 2n-dimensional F-vector space (F"")*© 
F'^ and let w be the standard symplectic form on Wn- The Heisenberg group 
Hn is the algebraic group with underlying algebraic variety Wn x F with the 
group law given by 

{wi,zi){w 2 ,Z 2 ) = (ici +W 2 ,zi + Z 2 + l/2u;(wi, W 2 ))■ 

Note that LLq = 1^- 

Definition 2.4.2. Let x be the additive character of F, as in ([7]). Extend x 
trivially to a character of the commutative subgroup 0 ©F*^ 0F C Hn- The 
oscillator representation Wy- is the unitary induction of x from 0 © F” © F 
to Hn- Define the smooth oscillator representation to be the space of 
smooth vectors in 

Lemma 2.4.3. cr^ = ind^j,„g 5 j,(x) 

Proof. In the archimedean case we apply the characterization of smooth 
vectors in a unitary induction given in [Pou72[ Theorem 5.1]. By this char¬ 
acterization ay. can be identified with the space 

{/ G C“((F^)*) G L‘^{{¥^)*)yi,j}. 

This space coincides with the Schwartz space 5((F”)*), which in turn can 
be identified with indQ^]p.„^p(x). 

In the non-archimedean case let us prove a stronger statement: ind^]p„^ji.(x) 
Ind^]p„i^p(x), where Ind denotes the full smooth induction. Indeed let 
/ G Ind^p„^]p(x), and let /' be the restriction of / to (F*^)* © 0 © 0. Since 
/ is smooth, i.e. fixed by an open compact subgroup K of Hn, for any 
If G (F"")* and u G F"" n R we have xipi'^)) f'(p) = f'ip)- This implies that 
/' has compact support, and thus / G ind^p„^]p(x)- C 

Theorem 2.4.4 (Stone-von-Neumann). The oscillator representation Wy 
is the only irreducible unitary representation of Hn with central character x- 

Corollary 2.4.5. Let L CW be a Lagrangian subspace. Extend x trivially 
to the abelian subgroup L © F C Hn. Then ind^^p x — cry. 

Lemma 2.4.6. LetW be non-archimedean. Let t be a smooth representation 
of Hn on which the center acts by the character x, cmd let r denote the 
smooth contragredient. Then (Hom//^((T;(-, r))* = Homj 7 ^(iT^,T). 
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Proof. Let L C W he a maximal lattice such that uj{L,L) C Ker(x)- 
Then by Theorem 12.4.41 < 7 .. = ind^^pX- By |BZ76l 2.29] Hom/f^((T 5 ^,T) = 
(r*)^. Since L is an open compact subgroup we get (r*)^ = (r^)* and 
RomH^{a^,T) = T^. □ 

2.5. Degenerate Whittaker models. 

Definition 2 . 5 . 1 . (i) A Whittaker pair is an ordered pair (S, (/?) such that 

S' G 0 is rational semi-simple, and ip G ( 0 *)!? 2 - Given such a Whittaker 
pair, we define the space of degenerate Whittaker models yVs,ip in the 
following way: let u := Define an anti-symmetric form on g 
by u}ip{X,Y) := p{[X,Y]). Let n be the radical of W(^|u- Note that u, n 
are nilpotent subalgebras of g, and [u, u] C g >2 C n. Let U := Exp(u) 
and N := Exp(n) be the corresponding nilpotent subgroups of G. Let 
n' := n n Ker((/ 9 ), N' := Exp(n'). If (^ = 0 we define 

( 8 ) Wsfi := indg(C). 

Assume now that p is non-zero. Then U/N' has a natural structure of 
a Heisenberg group, and its center is N/N'. Let denote the unitary 
character of N/N' given by Xi, 9 (exp(A)) := Let a^p denote 

the oscillator representation of U/N' with central character Xp^ 

(T'^p denote its trivial lifting to U. Define 

(9) Ws,p ■■= indg((Tp. 

(ii) Eor a nilpotent element G g*, define the generalized Whittaker model 

corresponding to p to be yVs,ip, where 5 is a neutral element for p 
ii ip ^ 0 and S = 0 if (/? = 0. We will also call yVs,ip neutral degenerate 
Whittaker model. By Lemma [2 .2.1 1 W,n depends only on the coadjoint 
orbit of p>, and does not depend on the choice of S. Thus we will also 
use the notation Wo for a nilpotent coadjoint orbit O C g*. In fJSjwe 
reformulate this definition without choosing S, but using the Killing 
form. 

(iii) Eor tt G M.{G) define the degenerate and generalized Whittaker spaces 
of TT by 

( 10 ) Ws,p{T^) ■= HomG(W 5 ,<^, 7 r*) and W<^( 7 r) := HomG(W<^, vr*). 

Note that Ws,= Homo(Ws,i^, t), where n denotes the contragredi- 
ent representation. In the non-archimedean case this is obvious and in the 
archimedean case this follows from the Dixmier-Malliavin theorem |DM78j . 

Lemma 2 . 5 . 2 . Let I C u be a maximal isotropic subalgebra and L := Exp([). 
Let TT G M{G). Then 

>V5,<^(7r) ^ Homi,(7r,x^^). 

Proof. By Corollary [233] and Lemma [2.3.2l we have 'Ws,^ — iHd£(x;^)- Using 
Lemma 12.3.41 we obtain 

= HomG(indf (x<^),7r*) ^ HomL(x<p, vr*) ^ HomL(7r, xi^^)- 

□ 
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In the case when F is non-archimedean and vr G J\A{G), slightly different 
degenerate Whittaker models are considered in |MW87j . Namely, let U" 
denote the subgroup of U generated by Exp(g^^) and the kernel of Xip- Let 
T^{U",xv) denote the biggest quotient of vr on which U" acts by the character 
Xp- Then [MW87j considers Hom[/((T^, By Lemma [2.4.61 and 

Frobenius reciprocity we have 

( 11 ) y^sA'^) - Oiomuia^,TT(u",xAy- 

Remark 2.5.3. For non-archimedean F we can define yVAis,ip to be the full 
induction Ind^cr^. Since for L as in Lemma 12.5.21 we have cr^ = Ind^ Xp^ 
our proof of Theorem will show that under the under the conditions of 
this theorem we have a G-equivariant embedding WAds^t^^WM-ip. For vr G 
M{G) one can define WMs,pA) HomG(vr, WA^s,^,). By the Frobenius 
reciprocity |BZ76[ Theorem 2.28] we have WMs,pA) ~ which 

by Lemma r2.5.2l is isomorphic to Ws,pA)- Thus, all the results of the paper 
can be reformulated in terms of the full induction. 

In order to have an analogous formulation in the archimedean case one 
needs a notion of full induction of smooth Frechet representations of moder¬ 
ate growth, that will satisfy transitivity of induction, Frobenius reciprocity 
(as in |BZ76[ Theorem 2.28]) and a'^ = Ind^ ^ certain full induction 
Om Ind^(7r, F) is defined in |dC191[ Definition 2.1.3]. It consists of func¬ 
tions of moderate growth from G to F which are equivariant under H. It 
satisfies the first two of our requirements but not the third one. Probably in 
the suitable notion of full induction the definition of function of moderate 
growth should take into account the action of H on F. 

Remark 2.5.4. If F is archimedean, one can define y^s,p for any semi-simple 
S with real eigenvalues in the same way, and the proof of Theorem [A] will 
be valid for this case without changes. 

3. Proof of Theorem El 

We will prove in H3.2I the following generalization of Theorem [A] 

Theorem 3.0.1. Let {S,A and {S,(p) be two Whittaker pairs in g such 
that (f G Gg(p. Suppose that n gf C gf ^ and that there exists a neutral 

element h for ip such that h commutes with S and S, and S — h commutes 
with (p. Then there is a G-equivariant epimorphism ofWs,p onto 

In order to deduce Theorem we will need the following lemma. 

Lemma 3.0.2 (See H3.ll) . Let V denote the set of conjugacy classes of 
Whittaker pairs in g and let Q denote the set of conjugacy classes of pairs 
of elements ip £ q* ,Z G gi^ such that ip is nilpotent and Z is rational semi¬ 
simple. 

Define a map p : Q ^ V in the following way: for any q G Q choose 
(Z, ip) G q, let h G Qz be a neutral element for and define fj,{q) to be 
the class of the pair {Z h, ip). Then the map p is a well-defined bijection. 

Proof of Theorem [31 By Lemma 13.0.21 there exists a neutral element h for 
ip which commutes with S. Then we have n A>i — 0 • Theorem 13.0.11 
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applied to the Whittaker pairs (/i, ip) and (5, ip) implies now that there exists 
a G-equivariant epimorphism of Wh,ip = onto y\^s,ip- C 

In the same way we obtain the following corollary of Theorem 1,3.0.11 for 
the case ip = ip. 

Corollary 3 . 0 . 3 . Let {S, ip) and (5, ip) be two Whittaker pairs with the same 
nilpotent element and commuting semi-simple elements. If hi 0>i h 
then there exists a G-equivariant epimorphism ofyVs,(fi onto 

Proof. By Lemma 13.0.21 applied to the group Gg_g we obtain that there 

exists a neutral element h for ip that commutes with S and S. Thus the 
corollary follows from Theorem 13.0.11 □ 

This corollary enables one to define a preorder on the set of models corre¬ 
sponding to a fixed nilpotent element ip. Let us describe this preorder more 
explicitly. Choose a neutral element h for ip and let o be a maximal split 
Cartan subalgebra in g that includes h. Choose a root system S on o. By 
Lemma [3.0.21 if [S, ip) is any Whittaker pair, then S is conjugate to h-\- Z 
for some Z in the stabilizer a<^ of ip in a. 

Definition 3 . 0 . 4 . Let X,Y G aip. We say that X Y if for any a gY 
such that a{h) < 0 and a{X) > 1 — a{h), we have a(T) > 1 — a{h). 

Corollary 13.0.31 immediately implies the following one. 

Corollary 3 . 0 . 5 . If X >(^ Y then there exists a G-equivariant epimorphism 
ofWh+x,ip onto Wh+Y,^- 

Remark 3.0.6. For ip ^ G(p, the condition on the existence of h cannot be 
omitted in Theorem 13.0.11 Indeed, let F be a p-adic field, ip he a regular 
nilpotent element in g* and 5 be a neutral element for ip. Let S = S. Then 
for any non-regular nilpotent orbit we can find a representative ip G Ggtp. 
However, for any supercuspidal representation vr of Gn we have Ws,^p{^^) = 0 
while Wg ^{tt) ^ 0 . 

Remark 3.0.7. The condition ip G Ggip in Theorem 13. 0.1 1 cannot be replaced 
by the weaker condition ip G Gip. Indeed, let G := GL(4,F), where F 

is a p-adic field. Let S := S := diag(3,1, —1, —3). Let ip,(p G (g*)! be 
defined by trace pairing with nilpotent elements in lower-triangular Jordan 
form with block sizes (2, 2) and (3,1) in correspondence. Then ip G Gip but 
/ ^ G-^ip. Let X be a character of GL(2,F), a be an irreducible cuspidal 
representation of GL(2,F) and vr := y x cj € Ai(G) be their Bernstein- 
Zelevinsky product. Then the spaces W 5 ,y,( 7 r) and Wg ^(vr) can be expressed 
through the Bernstein-Zelevinsky derivatives (see |BZ77j i in the following 
way: W 5 ,,^( 7 r) = D‘^{D‘^{'k)) and -(vr) = D^{D^{'k)). We have 

D\a) = 0, D\a) = C, D\x) = xIgl(i,q,), D\d\x)) = C, D\x) = 0, 
and by the Leibnitz rule for Bernstein-Zelevinsky derivatives 

D\tt) = X, D\d\7:)) = 0 , D3(vr) = xIgl(i.q,), D\d\7:)) = C, 
and thus Vyg^(vr) = C while 'Ws,ip{'^) = 0. 
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3.1. Proof of Lemma 13.0.2L We will need the following lemma. 

Lemma 3.1.1. Let {f,h,e) be an 5l2-triple in g, let L be its centralizer in 
G and let \ be its centralizer in g. Let Zi, Z 2 G I and suppose that h + Zi is 
conjugate to h + Z 2 by an element of Gf. Then Z\ is conjugate to Z 2 by an 
element of L. 

Proof. Note the Levi decomposition Gf = LU, where U is the nilradical of 
Gf. It is enough to show that u £ U, X £ Qh and ad{u)X £ Qh then 
ad{u)X = X. This holds since u = Exp(y) for some Y £ (gf) Cl g^lg and 
[Y,X]£Qh. □ 

Proof of Lemma\3fUf^ We choose a non-degenerate conjugation-invariant 
symmetric bilinear form on g and use it to identify g with g*. Thus, instead 
of G g* we will consider / G g. 

To see that p, is well-defined, let Z, f £ q. Let (/, h, e) be an s[ 2 -triple in 
Qz. Note that any two choices of such a triple are conjugate by Gz H Gf. 
Note also that for any g £ G, {ad{g)f,ad{g)h,ad{g)e) is an s[ 2 -triple in 
Qadig)z, and {ad{g)h + ad{g)z,ad{g)f) = ad{g){h + z,f) £ V. Thus p is 
well-defined. 

To see that p is onto, let c G P and (5, /) G c. Fix an embedding 
g C gl(y). Let us show that there exists a basis for V in which S is diagonal 
and / is in Jordan form. First of all let be all the eigenvalues of S, 

ordered such that Xj = A, — 2 only if j = i + 1, and let be the cor¬ 

responding eigenspaces. Then f(Wi) C Wi+i, and thus 
form a representation of a type A quiver. By |Gab72j (see also [BGP731 
Theorem 3.1(2)]), any such representation is a direct sum of indecompos¬ 
able representations in which all the spaces have dimensions 0 or 1. Each 
of these representations gives a Jordan chain for /. Then the union of these 
chains is the required basis. With respect to this basis / is in Jordan form 
and S is diagonal. Thus there exists a diagonal neutral element —h' for / in 
gl(I7), which then commutes with S. Since g is reductive, there exists a g- 
module projection p : gl(y)^g. Let h := p{h'). Then [/i, 5] = 0. Moreover, 
[h, /] = —2/ and h £ Im(ad(/)) and thus, by Lemma l2.2.1[ —h is a neutral 
element for /. Thus c = p{S — h, f). 

It is left to show that p is injective. Let q,q' £ Q such that p{q) = p{q'), 
and let (Z, /) and (Z', f') be their representatives. Then there exist s[ 2 - 
triples (/, h, e) in qz and (/', h', e') in g^/, and g £ G such that ad{g){f) = f 
and ad{g){h + Z) = h' + Z'. Note that {f, ad{g)h, ad{g)e) is an s[ 2 -triple 
and thus ad{g)h and h' are conjugate by Gf. Thus we can assume that 
f = f ,h = h', and h -|- Z is conjugate to h + Z' by Gf. By Lemma 13.1.11 
this implies that Z is conjugate to Z' by Gf and thus (Z, /) is conjugate to 
(Z',/'). ' □ 

3.2. Proof of Theorem 13.0.IL Let 00 denote the anti-symmetric form 

on g defined by lo{X, Y) := Tj). 

Our proof is based on the following lemma, which is in the spirit of 
[GRS991 Lemma 2.2] or [GRSlll Lemma 7.1] or [LaMaol^ Lemma A.l]. 

Lemma 3.2.1. Let I, r C g be nilpotent subalgebras such that [1, r] C 1 n r, 
a;|( = 0, u;|r = 0 and the radical o/a;|(+j is 1 n r. Then I -|- r is a nilpotent 
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Lie algebra and 

^^^Expd) — ^'^‘^Exp(t) 

Proof. If (/j = 0 then cj = 0, thus [ + r = [ n r, [ = r and there is nothing 
to prove. Now suppose ip ^ 0 and denote t ;= [ fl r fl Ker(X(^). Then 
Exp([ + r)/Exp(t) is the Heisenberg group corresponding to the symplectic 
form induced by uj on the space ([ + r)/([ n r). Since [/([ n r) and r/([ n 
r) are Lagrangian subspaces, the representations Xip and 

indg^p|[^y Xip are both isomorphic to the oscillator representation 

of Exp([ + r)/Exp(I) with central character defined by x<fi- Since Exp(t) 
acts trivially on Xip and indg^p||,^’^^ Xip^ we obtain that they are 

both isomorphic to the trivial extension of to Exp([ + r). □ 

Remark 3.2.2. By induction by stages we obtain ind^^p^^ Xtp — Xv^- 

Observe that this isomorphism can be realized explicitly as an integral trans¬ 
form: given / G iadg^pj-Q x^ we can define / G iadg^pj-^^ Xip simply by setting 

f{9) = [ f{ng) dn. 

J Exp([nr)\ Exp(r) 

Then the previous results imply that the map / eA / defines an isomorphism 
between these two spaces. 

In the course of our proof we will make several choices and introduce 
some notation. The reader is welcome to track those on Examples 13.2.111 
and 13.2.121 b elow. ^ 

Let 2 : := S — h and K := S — z. Choose a symmetric bilinear non¬ 
degenerate G-invariant form on g and let / G g correspond to p using this 
form. Let e be the nil-positive element for h and /. Then e G a := g^. 
Consider the embedding a* ^ g* corresponding to the bilinear form on g. 
Let A:=G^. 

Lemma 3.2.3. There exists ip' G ((a *)®)^2 such that p + p' ^ Ggp. 

Proof. Note that a* = (o*)®0O(i*(/)(a*). Since K preserves both summands 
we get 

(13) (a *)?2 = ((i .-)')?2 ffi {ad-ifmf, = ((a*)')fj ®od*(aK){v). 

Consider the map u : Ak x {{a*y )^2 {^*)-2 given by u{g, X) := g{p+X). 

Note that the differential of v at the point (1, 0) is onto, and thus the image 
of u contains an open neighborhood of i^(l,0) = p. Since p G Ggp, the 
image of v intersects the orbit G-^p. □ 

Let p' := p -\- p'. Let i denote the smallest of the /i-weights of p'. If 
(/?' = 0 we take i to be 0. Note that i is always non-negative. 

Lemma 3.2.4. If p' ^ 0 then there exists X G a<^ n af n such that 
p'{X) = 1. 

Proof. Let (/j' be the component of p' of weight i. There exists T G a with 
p'iiY) = 1. Let X' G 0 ^ be the component of Y in the decomposition 
a = [e, a] © a^p. Since p'^ G (a*)® , p'^^ vanishes on [e, a] and thus p'^{X') = 1. 
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Decompose a,p to joint eigenspaces of the commuting semi-simple operators 
h and K and let X be the component of X' in n n a^,-. Then cp'(X) = 

= ^'{x>) = 1. □ 

Let Z := S — S = K — h ^ Q^. For any rational number 0 < t < 1 define 

(14) St:=S + tZ, ut := ■= 0>i, and Wt := gf*. 

Definition 3.2.5. We call t regular if Ut = Ut+e for any small enough 
e € Q, or in other words tOt C Qz- If t is not regular we call it critical. For 
convenience, we will say that 0 is critical and 1 is regular. 

Note that there are only finitely many critical numbers. 

Lemma 3.2.6. The following results hold: 

(i) The form oj is ad{Z)-invariant. 

(a) Kerw = Q^ = gh c g<o- 
(Hi) Ker(w|,^,J = Ker(a;) n tUt. 

(iv) Ker(w|ut) = Ut © Ker(ti;|,x,t). 

(v) tVsCi Sip C Ut for any s <t. 

Proof. (1): oj{[Z,a],b)+u:{a,[Z,b]) = ip{[[Z,a],b])+(f{[a,[Z,b]]) = {ad*{Z){(f)){[a,b]) 

0. 

(pH) : a G Kerw uj{a,b) = 0V6 <^([o, 6]) = 0V6 {ad*{a){i^)){b) = 

0V6 -<=> ad*{a){if) = 0. Thus Kerw = q^. Since if is given by pairing with 
/, its stabilizer coincides with the space g^ that is spanned by the lowest 
weight vectors. 

(fnlll Ker(a;|n,J = tUi fl Kero; since uj{n3t, gg^) = 0 for any s ^ —1. Now 

Q<or\ix>t C gfo- 

(llv)l holds since Ut) = 0. 

m Let y G IDs n g^^ n gp. If p > 0 then Y G Ut. If p < 0 then 

Y G g>;^. By the conditions of the theorem we have g>]^ H gt^ C g>;^, thus 

y G 0>i ngfi C Ut. □ 

Choose a Lagrangian m C gf n gf and let 

(15) It ;= m + (ut n gfo) + Ker(w|uJ and tt := m + (ut n gfg) + Ker(w|ut). 

Lemma 3.2.7. (i) The spaces it and tt are ideals in Ut and [lt,tt] C ItHtt. 

(a) For any t >0, k and tt are maximal isotropic subspaces ofut. 

(Hi) Suppose that 0 < s < t, and all the elements of the open interval {s,t) 

are regular. 

Then C b. 

Proof, (ji]) follows from the inclusion [ut,ut] C Ot C li ©tt. 

(fii|l Since w is a(i(y)-invariant, we see that k and tt are isotropic. To show 
that It is maximal isotropic, let Y G gf H Ut. Let R denote the radical of 
cjIuj. If y € R then y G It. If s < 0 then Y G If. If s > 0 and Y ^ R 
then there exists Y' G g^g D Ut C b such that a;(y, y') ^ 0. Thus, if we 
enlarge b it will stop being isotropic. Now, note that uj defines a symplectic 
structure on Uj/i? ~ tDt/(tDt Llgg +tDtng(^), and the image of b in this space 
is Lagrangian. The image of tt is a complementary isotropic subspace, thus 
also a Lagrangian, and thus r* itself is maximal isotropic in Ut. 
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(fnUl Note that tUs n 0 ^q C C k- Let us show that 0 ^ C if Note that 
Os C Ut, since all the elements in (s,t) are regular. Let Y G 0 ^ be a joint 
eigenvector for ad{S) and ad{Z). If y ^ 0 * then y G roj and its Z-eigenvalue 
is negative. Thus y G h. Now by Lemma [3.2.6l([vj) we get lOsHg^ C C 

h, and from Lemma rs.2.61 (jii|l - (liv|) this implies Ker(a;|u^) C it- Altogether we 
get ts C it- □ 

Define 

(16) i[ := it n Ker((/ 3 ') and := r* n Ker((/ 9 '). 

Lemma 3.2.8. (i) For any 0 < s < t such that all the numbers in the 

open interval {s,t) are regular we have r' C !(. 

(ii) For 0 < t < 1 both [( and are subalgebras ofut, C H rj and 

for 0 < t < (z + l)/(f + 2 ) we have i[ = it and = tj. 

(Hi) [j n is the radical o/a;|[/^rJ 

Proof. Part (jl]) follows immediately from Lemma l3.2.7lliii]l . 

For part dn]) note that i > 0, G thus Ker((/?') D 

Thus, for t < 1 we have [ui,Uf] C Ker(yj') and for 0 < t < (z + l)/(z + 2) we 

have Uj C Ker((^'). Since [ut,Ui] C 0* C h fl we obtain (jii|) . 

For part ([m]) we can assume that t > (z + l)/(z + 2) and ip' / 0. Then, by 
Lemma [3.2.41 there exists X € Hut with ip'{X) = 1. Thus, the image of 
in Ui/( 0 ,^ nuj) coincides with the image of it, and the image of coincides 
with the image of Xt ■ Since those images are Lagrangian, n is the radical 
of WI [/+,/. □ 

Lemmas 13.2.81 and 13.2.11 imply that 

(17) ~ indf^p(^,) mdf^p((,) Xv^ mdf^p(^,) Xy. 

and for s < t such that all the numbers in (s, t) are regular we have 

(18) ™dExp(rD “df^p(l') 

We are now ready to prove Theorem 13.0.11 

Proof of Theorem \3. 0.li Let 0 < ti < • • • < be all the critical numbers. 
Note that x'^^ is an isotropic subalgebra of Ui, and that it is also isotropic with 
respect to the form u;<^+(p/(A, y) := (ip + ip'){[X,Y]). Let q be a maximal 
isotropic subspace of Ui with respect to this form. It includes Oi and thus 
is necessary a subalgebra. Note that 

(19) indf^p(,, j Xv. = ™df^p(r'J “^Exp(q) X^+v' ^ 

Thus 


(20) Ws,^ ^ ™^Exp(.J,) ) Xv. - “df^p(^,^) 

“Cp(i; J Xv. - J X^-Wg_-. 

□ 

Remark 3.2.9. Observe that, in the above proof, the map ind^^p^j./ j ™‘^Exp(iJ, ) Xp 
is simply given by integration over Exp(r(J\Exp([(.^^). From this, and the 
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observation given in Remark 13.2.21 we see that the G-equivariant epimor- 
phism promised in Theorem 13.0.11 is given by a series of integral transforms 
followed, if necessary, by conjugation by an element of G. 

Remark 3.2.10. Let M denote the joint centralizer of ip, (p, S, S, and h. Then 
a central extension M of M acts naturally on the oscillator representations 
a in and ( 7 < 5 , and thus also on the degenerate Whittaker models Wsw and 


It is easy to see that the constructed epimorphism W 5 ,in¬ 
tertwines these actions. 


- in- 

S,ip 


Let us now present two examples for the elements and subalgebras dehned 
in the course of the proof. Let G := GL(4,F) and dehne (p by <p{X) := 
Tr(X(Ll 2 i + L' 43 ))) where E 21 and £"43 are elementary matrices. Let h be 
the diagonal matrix diag(l, —1,1, —1) and S := h. 

Example 3.2.11. Let (p := ip, K := diag(3,1, —1, —3), 2 ; := 0. Then 
Z = diag(2, 2, —2, —2), St = diag(l -|- 2t, —1 + 2t,l — 2t, —1 — 2t) and the 
weights of St are as follows: 


/ 0 
-2 
-At 

\ -At-2 


2 

0 

-At+ 2 
-At 


At 

At-2 
0 

-2 


4t + 2 \ 
At 
2 

0 / 


The critical numbers are 1/4 and 3/4. For t > 3/4 we get the principal 
degenerate Whittaker model. We have = tj, [^ = h for all t. The above 
system of inclusions of to C li/4 ~ ri/4 C [3/4 = r3/4 is: 


/ 0 

- 0 

-V 


/ 0 

— 

a 

-V 


/ 0 

— 

* 

-V 

/ 

0 

0 0 

0 

c 

0 

0 

0 

a 


0 

0 

0 


c 

0 

- 0 

— 

0 

* 

0 

— 


0 

0 

0 

— 

VO 

0 0 

0 ^ 


VO 

0 

0 

0 ^ 


1 0 

0 

0 

0 ^ 

V 


0 

0 


0 * 
0 0 
0 0 


Here, both * and — denote arbitrary elements. — denotes the entries in Oj 
and * those in tUt. The letter a denotes an arbitrary element, but the two 
appearances of a denote the same numbers. The passage from [ 4/4 to ri /4 is 
denoted by ~. At 3/4 we have [ 3/4 = r 3 / 4 . 

Let us now give an example in which p and <p are not equal and not 
conjugate. 

Example 3.2.12. Identify 0 0* using the trace form and let 



( 0 

0 

1 

0 V 


( 0 

1 

0 

0 V 


( 0 

0 

1 

0 \ 


1 

0 

0 

1 


0 

0 

0 

0 

/ 

0 

0 

0 

1 

p = 

0 

0 

0 

0 

, e = 

0 

0 

0 

1 

, ^ = 

0 

0 

0 

0 


VO 

0 

1 

0^ 


VO 

0 

0 

0^ 


VO 

0 

0 

0/ 


Let S := diag(0, —2, 2,0), Z = diag(—1, —1,1,1). Note that tuo = tui = 0 
and the only critical value of t is 1/2. The sequence of subalgebras from the 
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proof of Theorem 13.0.11 is 



/ 0 

* 

0 

- \ 


/ 0 

* 

0 

0 V 


/ 0 

* 

0 

0 \ 

7 / if 

0 

0 

0 

0 

! 

0 

0 

0 

0 


0 

0 

0 

0 

Uq = to = U/2 ~ 

0 

* 

0 

* 

~ H/2 = 

a 

* 

0 

* 

c 

* 

* 

0 

* 


VO 

0 

0 

0 ^ 


\ 0 

—a 

0 

0^ 


VO 

* 

0 

0 / 


3.3. Principal degenerate Whittaker models and proof of theorem 

m In the discussion below o will denote a maximal split toral subalgebra 
of 0 , we will write S(a, g) for the corresponding (restricted) root system, 
S'*' (a, g) for a choice of positive roots, and A (a, g) for the corresponding 
system of simple roots. 

Definition 3.3.1. We say that a rational semisimple S' G g is principal 
if there exists an a containing S, and a simple subsystem A = A (a, g) C 
S (a, g) such that a (S) = 2 for all a € A. We say that a Whittaker pair 
(S', ip) is principal if S is principal. A principal degenerate Whittaker model 
is the degenerate Whittaker model corresponding to a principal Whittaker 
pair. 

We fix a non-degenerate invariant bilinear form on g, which allows us to 
identify nilpotent elements in g and g*. Thus we may equally well apply 
the above terminology to “Whittaker pairs” (5, /) C g x g, such that S is 
rational semisimple and [S', /] = —2/. 

Note that a principal element S uniquely determines both o, and the 
simple system A = A(a, g). Thus there is a bijection between principal 
elements and simple systems which we denote by S' i—)• A 5 , A i->- 5 a. 

If (5, /) is principal then setting A = A 5 , we can write / uniquely in the 
form 

( 21 ) / = • 

for some G 0 -a- Conversely if / is of the form (| 2 ip for some A, then we 
will say that / is a PL nilpotent and A is compatible with /. In this case 
(5a,/) is a principal Whittaker pair, and we define the A-support of / to 
be 

suppA (/) = {a G A I Yq, / 0} . 

If suppA (/) = A we say that / is a principal nilpotent element. Note that 
this notion is weaker than the similar notion defined in |Bou75[ VIII.11.4]. 
However, if G is quasi-split then both notions are equivalent to the notion 
of regular nilpotent element. 

Note that / is a PL nilpotent if and only if it is a principal nilpotent 
element for a Levi subgroup of G. For the general linear groups, every orbit 
includes such an element. For complex classical groups, all such orbits are 
described in [GS15[ § 6 ] in terms of the corresponding partitions. For complex 
exceptional groups, these are the orbits with non-parenthetical Bala-Carter 
labels. 

Lemma 3.3.2. Let (5i,/) and ( 52 ,/) be two Whittaker pairs with the same 
f, and let h be neutral for f. Then there exist 51,52 € G^ and Zi,Z 2 G 
0 /i C 0 j such that 

gi ■ Si = h + Zi, 52 • 52 = h -|- Z 2 , [Zi, Z 2 ] = 0 . 
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Proof. By Lemma [3.0.21 we can find gi, § 2 , Zi, Z 2 satisfying all the stated 
properties, except perhaps the commutativity [Zi,Z 2 ] = 0. Since H gj 
is the centralizer of an s[ 2 -triple, it is reductive. Since Zi,Z 2 € 0 h Ll gj are 
rational semisimple, we can move them to a common Cartan subspace after 
further conjugation by elements of fl G/. □ 

Proposition 3.3.3. Let {S, /) be a Whittaker pair such that f is a PL 
nilpotent. Then there exist a maximal split toral subalgebra a, and a simple 
system A C C S (a, g) such that 

(a) A is compatible with f. 

(b) a contains S. 

(c) a contains a neutral element h for f. 

(d) If a £ a satisfies a{h) < 0 and a{S) > 0 then a € 

Proof. Let us first choose any simple system A (b, g) compatible with /. By 
Lemma 13.3.21 there exists g in the centralizer Gf oi f such that g ■ S € b. 
Now the action of g~^ carries b to a maximal toral subalgebra a and A (b, g) 
to a simple system A' = A' (a, g) which satisfies (a) and (b). Next note that 
if S' = 5a' then {S',f) is a Whittaker pair, so by Lemma [3.0.21 there is a 
neutral element h for / that commutes with but this forces h £ a and 
thus (c) holds. 

If suppi\i (/) = A' then we set A = A'; in this case we have h — S is 
central and (d) holds vacuously. Now suppose supp/\i (/) C A'. Then we 
will show how to modify A' to obtain a new system A ( 0 , g) such that (a)-(d) 
are satished. For this let us write 

Z = S-h, Z' = S'-h, he = Z + eZ' + e^h. 

where e > 0 is chosen sufficiently small so that for all a G S (a, g) we have 

(22) a{Z)>0 a{Z) > e\a {^Z'^\ + \a{h)\ 

(23) a{Z') >0 ^ a{Z') > s\a{h)\ 

We claim that h^ is regular in the sense that a {hs) 7 ^ 0 for all a. If a (Z) > 0 
this follows from (j22l) . if a (Z) < 0 we simply replace a by —a. If a {Z) = 0 
but a {Z') 7 ^ 0, then this follows analogously from (I23p . Finally, if a (Z) = 
a (Z') = 0 then a {h^) = {h) and we must show that a {h) 7 ^ 0, but this 

follows from the regularity of S' = h + Z'. 

This means that we can define a positive root system as follows 

(a, g) = {a G S I a {h^) > 0} . 

Let A be the corresponding simple system; we will show that supp/\i (/) C 
A. This implies that A is compatible with e, so that (a) holds. To prove that 
supp^i (/) C A suppose a belongs to supp^^i (/). Then we have a{Z) = 
a {Z') = 0, a (he) = (h) = 2e^ > 0 This means that a £ (a, g) and 

it remains to show that we cannot write 

(24) a = /3 + y 

where /3 ,7 G (a, g). Now if /3 (Z) > 0 or /3 {Z') > 0 then is impossible 
by ([2^ and (l23l) . Thus we may assume /3 (Z) = P {Z') = 0 and thus 
/3 (S') = /3 {Z') + f3 (h) = 2 > 0, and similarly 7 (S') > 0. But a is a simple 
root in the positive system defined by S', so (IMl) cannot hold. 
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Finally we verify that (a, fl) satisfies (d). Thus suppose a € S ( 0 , 0 ) 
satishes a (h) < 0 and a (S) > 0, then we have a {Z) = a (S) — a {h) > 0. 
By (12^ . this implies a (h^) = a {Z) + ea {Z') + e^a {h) > 0. Thus we have 
a € (a, 0 ) as desired. □ 

Proposition 3.3.4. Suppose that (/? G 0 * can he completed to a principal 
Whittaker pair. Then any degenerate Whittaker model yVs,^p can be mapped 
onto some principal degenerate Whittaker model 

Proof. By Corollary 13.0.51 it suffices to hnd a maximally split toral subalge¬ 
bra a and containing a neutral element h for ip, and a principal element S 
such that S — h ^ and S — h >^p S' — h in the sense of Dehnition I3.U.4[ 
i.e. 

a{h) < 0 and a (S) > 1 a > 1 for all a G S (a, 0 ). 

Let / G 0 be the nilpotent element corresponding to ip. Then (5, /) sat¬ 
ishes the conditions of Proposition 13.3.31 Let a, h, A be as in the propo¬ 
sition, and let S = S'a- Then any a G S (a, 0 ) with a{h) < 0 and a (S') > 1 

lies in S"''. Since S is principal, we deduce that a i S] > 2. □ 


3.3.1. Archimedean case. For a smooth representation tt of a real reductive 
group G one can dehne one more invariant, which we denote V( 7 r) and call 
the annihilator variety of tt. It is sometimes called the associated variety of 
the annihilator of vr. It is dehned to be the set of zeros in 0 ^ of the ideal 
in the symmetric algebra S( 0 c), which is generated by the symbols of the 
annihilator ideal of tt in the universal enveloping algebra 17 ( 0 c)- It follows 
from |Vog91 Theorem 8.4] and [SV0n| that V(' 7 r) is the Zariski closure of 
WF( 7 r) in 0 ^.. Note that if G is a complex reductive group or G = GL„(M) 
we have WF( 7 r) = V( 7 r) n 0 *. 

We will use the following theorems 


Theorem 3.3.5 l |Mat87[ Corollary 4]). Let tt be a smooth representation 
of G, let O C Q* be a nilpotent orbit and suppose that VV’o( 7 r) 7 ^ 0. Then 
O C V( 7 r). 

Theorem 3.3.6 1 |CS15[ Theorem Bj). Let (5, (/?) he a principal degenerate 
Whittaker pair for G. Let tt G Ai(G) such that p G WF( 7 r). Then 
(i) If G is a complex group or G = GL„(M) then VV’s,(^( 7 r) 7 ^ 0. 

(a) If G is quasisplit then there exists g G Gc such that ad{g) preserves 0 
and ')Wad{g){S),ad{g){ip){^) 7^ b- 

In fact, the theorems in |Mat87[ lGS15j are stronger than the versions we 
state here. 


Proof of Theorem\^ Let (S', p) be as in the theorem. Then p can be com¬ 
pleted to a principal Whittaker pair, and Proposition 13.3.41 implies that the 
degenerate Whittaker model yVs,tf can be mapped onto some principal de¬ 
generate Whittaker model By TheoremIXl W.n maps onto ,n. Thus 

we have ^( 7 r)<^>Vs,(^( 7 r)‘— 7 -VV<^( 7 r). Together with Theorems 13. 3. 513. 3. 61 

we get 

p G WF(7r) ^ Wg_^(7r) / 0 ^ ^ 0 ^ / 0 ^ G WF(7r). 
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□ 

In the same way we get the following statement for real reductive groups. 

Corollary 3.3.7. Suppose that G is quasisplit and let vr G A4(G}. Let 
(p G WF( 7 r) he a PL nilpotent, and let {S,ip) be a Whittaker pair. Then there 
exists g G Gc such that ad{g) preserves 0 and ')Wad{g){s),ad{ 9 ){^)^'^') ^ 

4. General linear groups 

4.1. Notation. Let us first introduce some notation. A composition g oi n 
is a sequence of natural (positive) numbers r/i,..., % with YlVi = n. The 
length of g isk. A partition A is a composition such that Ai > A 2 > • • • > A^. 
For a composition g we denote by g- the corresponding partition. A partial 
order on partitions of n is dehned by 

j j 

(25) X > fi Aj > for any I < j < length(A), length(//). 

7=1 i=l 

We will use the notation diag(xi,..., x^) for diagonal and block-diagonal 
matrices. For a natural number k we denote by G 0 ^ the /oirer-triangular 
Jordan block of size k, and by hk the diagonal matrix hk ■= diag(A: — l,k — 
3,... ,1 — k). For a composition g we denote 

(26) Aj .— diag(,..., ) G Qn and .— diag(hjjj j...; hrjf .) G Qn- 

Note that [hrj, Jg] = —2J,j and {Jrj, hrj) can be completed to an 512-triple. 

Identify 0 * with 0 „ using the trace form. Denote by Orj the orbit of 
Jr^, and also the corresponding orbit in 0 *. By the Jordan theorem all 
nilpotent orbits are of this form. It is well known that Or^ C 0.y if and only 
if g- < 7 -. Let C Gn denote the subgroup of diagonal matrices and 
t„ C 0 n the subalgebra of diagonal matrices. 

4.2. Proof of Theorem [Dl Let Eij denote the elementary matrix with 1 
in the (i,j) entry and zeros elsewhere. 

Lemma 4.2.1. For any p,q,r G Z>o with p > r there exists a diagonal 
matrix S G tm(^), where m = p + q + r, and a regular nilpotent X G 0g+r 
such that [5*) T^p+q,r)] — ‘^J{p+q,r) diag(Jp, A) G iGm)sJ[p+q,r)' 

Proof. If r = 0 or 0 = 0 we take X := Jq+r, S := and note that 

diag(Jp, A) G TmJ(p+q,r) C {Gm)sJ(p+q,r)- Assume now g, r > 0 and let 
F := -S' := diag(hp+g, hr + {r + q-p) Id,.) G 0m, 

and g .— (Idm TF),—r+l,m—r+l)(Idm TFp—r+ 2 ,m—^+ 2 ). (fd-m ~\~Ep,m) S Gm- 

Note that Si := Su = p + q — {2i — 1) for 1 < i < p + q and 5, = 2r + 
3q + p — {2i — 1) ioY p + q + 1 < i < m. Thus Sm-r+j = Sp-r+j for 
1 < J < I" and thus g commutes with S. Note also that F' := Ad{g){F) = 
F + Conjugating F' by a suitable diagonal matrix we can obtain 

F' — (1 — t)Fp_|_i^p G {Gm)sF for any t G Letting t go to zero, we get 

that _ 

f ■= P' — Ep+l,p = F + Fp_|_i^m ~ Fp_|_i^p G (Gm)sF. 

Finally, it is easy to see that / = diag(Jp, A) for a regular nilpotent A G 

Qq+r- FI 
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Lemma 4.2.2. Let X, ^ he partitions of n with A > /i. Then there exists 
an index i < length(A) such that Xi > fj,i > Aj+i. Here, if i = length(A) we 
take Aj+i = 0 . 

Proof. We prove by induction on length(A). If length(A) = 1 take i = 1. 
For the induction step, assume length(A) > 2 and the lemma holds for all 
shorter partitions. If p,i > X 2 take i := 1. Otherwise, consider the partitions 
X' = (Ai + A 2 — /ii, A 3 ,...) and p! = {p 2 , P 3 , ■ ■ ■)■ Note that these are indeed 
partitions and A' > p'. Thus, by the induction hypothesis there exists j 
such that X'j > p'j > A'-^^. If j > 1 take i := j + 1. If j = 1 then p 2 > A 3 
and we also have X 2 > pi > P 2 - Thus we can take i := j +1 = 2 in this case 
as well. □ 


We are now ready to prove Theorem |Dj Let us reformulate it in terms of 
partitions. 

Theorem 4.2.3. Let A, p he partitions ofn. Then X> p if and only if there 
exists S G tn(^) such that [5, Ja] = —2Ja and {Gn)sJx intersects 

Proof. We prove the lemma by induction on re. The base case re = 1 is 
obvious. For the induction step, assume that the lemma holds for all re' < 
re. By Lemma 14.2.21 there exists an index i with i < length(A) such that 
Xi P Pi P A 2 - 1 - 1 . Let 

r := Aj+i, q:= Pi - Aj+i, p := A* + Aj+i - pi, and rre := A* + Aj+i. 

Now consider the partition X' obtained by replacing the blocks Xi and Aj+i 
by a single block p, and the partition p' obtained from p by omitting the 
block Pi. Note that both are indeed partitions oi n — pi and that X' > p'. 
Thus, by the induction hypothesis there exists S" G (Z) such that 
[5", Ja'] = —2Jx> and {Gn-in)s" Jx' intersects O^'. Choose S' G tm(^) and 
X G Qfjn using Lemma 14.2.11 Consider the matrix Z' formed by taking the 
first p elements on the diagonal of S' and the matrix Z" formed by taking 
the p elements number Ai + • • • + Aj-i + 1 ,... , Ai + • • • + Aj-i + p on the 
diagonal of S". Note that Z' — Z" is a diagonal matrix that commutes with 
Jp and thus equals c Idp for some integer c. Replacing S' by S" — c Idm we 
can assume that Z' = Z" and thus there exists S G t„(Z) that includes both 
S' and S" as diagonal submatrices. 

Let us show that S satisfies the conditions of the lemma. Let a := Aj 

and d := • Define an embedding of Gm into Gn by Li{g) := 

diag(Ida, 5 , Idrf). Dehne an embedding of Gn-^n into by 



'n-fii 

into 

0 

B 

Ida 

0 

0 

D 

',F), 

G G 


Mat{d + p,d + p,¥). Let dii : Qm^Qn and dL 2 ■ Qn-fii^Qn be the dif¬ 
ferentials of ii,L 2 - These embeddings map the centralizers of S' and S" 
into the centralizer of S. Let Y := (ii-i(diag(0, X)) + dL2{Jx') £ 0n- Since 


diag(Jp, G {Gm)s'•J{Xi,Xi. 


+1 


) , we have Y G {Gn)sJx- Since {Gn-pi)s"Jx' 

□ 


intersects Ow, {Gn)sY intersects Op and thus {Gn)sJx intersects O 


u- 
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4.3. Proof of Theorem [El In the archimedean case the theorem follows 

from Theorem 13.3.51 and Corollary 13.3.71 Thus we assume here that F is 
non-archimedean. Let (/9 G g* and let u : ^ G be an algebraic group 

morphism (defined over F) such that ad*{u{t))ip = Let S ;= du{l) G g. 
Following |MW87] define := Ws,^- 

Theorem 4.3.1 f |MW87[ Proposition 1.11, Theorem 1.16 and Corollary 
1.17], and |Varl4[ Proposition 1 and Theorem Ij). Let (p,u be as above. Let 
7rGM{G). 

(i) / 0 then p G WF( 7 r). 

(a) If p belongs to a maximal orbit O G WFC( 7 r) then / 0 and 

its dimension equals the coefficient of O in WFC( 7 r). 

Proof of Theorem fS Let vr G M.{Gn). Theorem l4.3.1l imDlies that if WoCt) 7 ^ 
0 then O C WF( 7 r). Suppose now that O C WF( 7 r), i.e. there exists 
O' G WFC( 7 r) such that O C O'. Let p G O' and S = diag({/ij}) G gl„(Z) 
be as in Theorem ini Define u : F^ by n(t) := diag(t^»). Then 

ad*{v{t))p = t~‘^p and Wu,p = By Theorem 14.3.11 Wg 7 ^ 0 and 

by Theorems lAl and iDl we have an epimorphism Wo^y^s,(p-, hence 'Ws,p{t^) 
embeds into Wc)( 7 r), and thus 7 ^ 0 . □ 

Remark 4.3.2. One can show that if p is obtained from A by taking some 
parts apart, or by replacing two parts of the same parity by two equal 
parts, then maps onto Wox ■ This follows from Theorem El by taking 
S := hx, p{X) := Tt{XJx), and p{X) := Tr(XJ^). However, this does not 
extend to arbitrary p < X. For example, if A = (4,1) and p = (3, 2) then 
( 0 n )^2 does not intersect 

For the symplectic groups one can show that if p is obtained from A by 
replacing two parts of the same parity by two equal parts, then one can map 
the generalized Whittaker model corresponding to an orbit with partition 
p onto the generalized Whittaker model corresponding to an orbit with 
partition A. 

4.4. Definition of derivatives and proof of Theorem [Fl The notion 
of derivative was first defined in |BZ77j for smooth representations of Gn 
over non-archimedean fields and became a crucial tool in the study of this 
category. In [AGS15aj this construction was extended to the archimedean 
case. 

The definition of derivative is based on the “mirabolic” subgroup Pn of 
Gn consisting of matrices with last row (0,..., 0,1). The unipotent radical 
of this subgroup is an (n — l)-dimensional linear space that we denote Vn, 
and the reductive quotient is Gn-i- We have a natural isomorphism Pn = 
Gn-i IX Vn. The group Gn-i has 2 orbits on I 4 and hence also on the dual 
group V*'. the zero and the non-zero orbit. The stabilizer in G„_i of a 
non-trivial character of Vn is isomorphic to Pn-i- 

Let ^|Jn be the standard non-trivial unitary character of Vn, given by 

Ipnixi, ..., Xn-l) := x{Xn-l), 

where x is the fixed additive character of F, as in (|7]). We will also denote 
by ifn the corresponding character of the Lie algebra For all n and for 
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all smooth representations tt ol Pn, we define 

T^iVn,ipn) ■= Span{7r(a)i; -'ipn{a)v : v Gtt, ae K}, 


and we put 
(27) <i>-(7r) 


71/71(14, V’n) 
77/71(14,-i/n) 


if IF is archimedean 
if IF is non-archimedean 


If F is non-archimedean, our definition of coincides with the one in 
|BZ76l §5.11]. It differs from the definition in |BZ77] by the twist by the 
character | det | . 

For a smooth representation tt of Gn we define a representation E^{'k) of 
Gn-k by 

(28) £‘(,r):=((<l.-)‘-'(,r|pJ)lG.. 


We call it the k-th pre-derivative of tt. 

Define also a functor : Rep°°(i4) —>■ Rep°°(Pn+i) by 

(29) (tt) = indpl+^y^^^ (tt K i/n+i ), 

where Rep°° denotes the category of smooth representations as in Definition 
12.3.11 and : Rep°°(G„) Rep°°(G„+fc) by 

(30) iHir) := ind4y(<|.J)‘-'(indg«- p)). 

Lemma 4.4.1. Let X be a partition of n and let tt G A4(Gn)- Then 

.. (IF^Htt) ... )))4 


The proof of this lemma is analogous to the proof of Lemma 12.5.21 
If F is archimedean then the space 7r(14,V’n) has the same closure as 
the space 7 r(t)„, 7 /n) defined analogously using the Lie algebra action, since 
both closures equal to the joint kernel of all (I 4 , 'i/’n)-equivariant continuous 
functionals on tt. It is shown in |AGS15b] that if tt G Ai{Gn) then 71 ( 0 ^, i/n) 
is closed. Moreover, it is shown that for any i, the space (<I>“)*(7r)(Dn, i/n) is 
closed in this case. Thus for tt G A4(Gn) our definition of E^ coincides with 
the functor E^ used in |AGS15al IAGS15b] . It differs from the functor E^ 
used in |AGS15al IAGS15bj by the twist by the character | det Note 

though that for non-admissible smooth tt our definition of E^ might differ 
from the functor E^ used in |AGS15a[ IAGS15bj . 

Let us now start proving Theorem |Fl Let A be a partition of n and rj be 
the inverse reordering of A. Let / ;= J^, h := and let e G 0 n be the 
unique element such that (/, h, e) is an s [2 triple. We will prove Theorem 
[F] by induction on k := length(r 7 ). Let Z be a diagonal matrix with first 
n — rji^ entries equal to zero, and last rj^ entries equal to % + %_!. Define 
(/? G 0 * by ipiX) := Tr(/A) and let w := uj^p. For 0 < t < 1 let := h + tZ 
and define Ut, k and r* as in formulas (II4II5II in T3.2I (note though that our 
[(,r( differ from the ones in formula (fTHil 'l. Let 0 denote the stabilizer of the 
standard basis vector and define [( := a n h, := a fl r*. 

Lemma 4.4.2. (i) and are subalgebras of Ut and C nr(. 

(a) Uo C a and thus Iq = Ig. 
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Proof. Part ([I]) follows from Lemma I3.2.71II1) and the fact that o is a subal¬ 
gebra of 0 . 

For part ([ill) note that the /i-weight of e^-r^fe+i is the maximal weight 
inside the standard representation, and thus any element of Uq annihilates 
it. □ 

The next lemma follows from the structure of the lowest-weight vectors 
in a tensor product of irreducible representations of s[ 2 . 

Lemma 4.4.3. Let a and r be two irreducible representations of an s[ 2 - 
triple {f,h,e) with dimcj > dimr. Let L := HomF((T, r) = a* Let L^ 
denote the annihilator in L of the highest-weight vector in a, and L^ denote 
the space spanned by all the lowest-weight vectors in L. Then L = L^ (B L^. 

Lemma 4.4.4. u* = Ut n a© 

Proof. First decompose Ut = (ut)<Q © (ut)^Q and note that (uj)<q C Uq 
and thus {vit)<Q C o and (ut)fQ n {ut)ip C (uq)'^ = 0. Now let V denote the 
standard representation of 0n and consider the decomposition V = Fl © • • • © 

Vk, where each F is the subspace spanned by the basic vectors with indices 
from 1 + "dj to 'dj- Note that F is an irreducible representation 

of the s[2 triple (/, h , e) of dimension pi . Then (uj)>q = ©(©/ Hom]F(Ffc, F)- 
Bv Lemma 14.4.31 we have = (ut)fona©((u*)foV. □ 

Corollary 4.4.5. The radical of the restriction is Ij Hr^. 

Proof. By Lemma [3.2.7l|ii]l we have Rad(w|(j+rJ = hntt and thus Rad(a;|(/+tj) D 
nrj . Now, (ut)^ C [j nr^ and by Lemma [4.4.41 the image of in Uj/(ut)<^ 
coincides with the image of h and the image of in uj/(ut)<^ coincides with 
the image of Xf Thus Rad(u;|(/_,_j/) = Ij n rj. □ 

Recall Definition 13.2.51 of regular and critical numbers. 

Lemma 4.4.6. For s < t with no critical numbers in the open interval {s,t) 
we have = Ij. 

Proof. By Lemma r3.2.7Hiiip we have C it and thus C For the inverse 
inclusion note that 

[[ = m © (t)t)fo © (tit)fo ® (tt’t)fo- 

We have m C Os C r^. Since there are no critical 

elements in (s,t) we also have ® (Us)fo ® ° 

Proof of Theorem O Corollary 14.4.51 and Lemmas 14.4.61 and 13.2.11 imply 

(31) - indg):p((o)(w) - ™d^;p(rl)(llv^)- 

Let r]~ := (r/i, ..,r]k-i) and consider the corresponding elements h^- G Qn-Tjk 
and G 9n-r]k ■ Note that 

(32) in4x"pw)(^^) ^ 

The isomorphism dH) now follows by induction on length(r/). The isomor¬ 
phism ([5]) follows from ([4]) using Lemma 14.4.11 □ 
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4.5. Proof of Corollary O For non-archimedean F, exactness is well- 
known, the rest of part (|I]) follows from Theorem 14.3. 11 and part (jn|) follows 
from |MW87l §11.2]. Thus we assume that F is archimedean. 

4.5.1. Preliminaries on pre-derivatives. The highest non-zero pre-derivative 
of vr G M.{Gn) plays a special role. It has better properties than the other 
derivatives. In particular it is also admissible. The index of the highest non¬ 
zero pre-derivative is called the depth of vr. As shown in [GS131 IAGS15al 
IGS15j the depth also equals the maximum among the first parts of the 
partitions in the orbits in WF(7r). The following theorem summarizes the 
main results of |AGS15al IAGS15b] . 

Theorem 4.5.1. Let A4'^{Gn) C A4(Gn) denote the subcategory of repre¬ 
sentations of depth < d. Then 

(i) The functor : A4{Gn) Rep°°{Pn _fc+i) is exact for any 1 < k < n. 

(a) defines a functor M.'^{Gn) M.{Gn-d)- 

(Hi) Let n = ni + ■ ■ ■ + Ud and let Xi be characters of Gm- Let vr = xi x 
■ ■ ■ X Xd £ AI'^(Gn) denote the corresponding monomial representation. 
Then 

= ((xi)|g„i_i X ■■■ X (yd)|G„^_i) 

(iv) If T is an irreducible unitary representation of Gn and has depth 
d then E‘^{t°°) = where At denotes the (irreducible, unitary) 

adduced representation defined in |Sah89| (cf. |Bar03j ). 

For non-archimedean F, the theorem follows from |BZ77] since E'^ coin¬ 
cides with the highest Bernstein-Zelevinsky derivative considered in |BZ77j . 

For archimedean F, WF(£'^(7r)) is calculated in [GS15] . In particular, 
|GS15[ Theorem 5.0.5] implies the following result. 

Theorem 4.5.2. Let F be archimedean and let vr G A4(G„). Suppose 
that WF(7r) = ni > ... > n^. Then depth{'n) = ni and 

WF{E^^{7r))=On,’..,n,. 

4.5.2. Proof of Corollary Let A be a partition of n and 0\ be the corre¬ 

sponding nilpotent orbit. Denote VV’A(vr) := {E^><^{... E^^{tt) ... ))*. We use 
Theorem [Fl and identify Woxi'^) with VV’A(7r). We prove the theorem by 
induction on n, using Theorems 14.5.II and 14.5.21 For the base of the induc¬ 
tion we note that A4(Go) is the category of finite-dimensional vector spaces, 
and monomial or irreducible representations of Gq are one-dimensional. For 
the induction step, let /x := (A 2 ,..., Afc) and note that by Theorem 14.5.11 
E^^ is an exact functor from (Gn) to A4(Gn-\i). By Theorem 14.5.21 

it maps M.-^^(Gn) to A4-‘^>"(Gn-Xi), and E^^(7r) G A4''^'"(G„_Ai) if and 
only if vr G A4"^^^(Gn). Thus, by the properties of quotient categories (see 
|Gab62[ §III.l]), E^^ defines an exact and faithful functor from A4^^(Gn) 
to Theorem 14.5.11 also implies that if vr G A4-^^(Gn) is 

monomial then so is E^^(Tr) and if vr is irreducible unitarizable, then so is 
E^^ (vr). By the induction step, kV^ defines an exact and faithful functor from 
M^i^iGn _Ai) to the category of finite-dimensional vector spaces, and 
maps monomial representations and irreducible unitarizable representations 
to one-dimensional spaces. Thus, so does Wa = o E^^. □ 
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5. Choice-free definitions 

5.1. Generalized Whittaker models. In this section we define the gener¬ 
alized Whittaker model corresponding to a nilpotent element e € g, without 
choosing a neutral element h. First of all, the filtration g>fc (unlike the grad¬ 
ing Qk) can be defined without choosing h. It is in fact called the Deligne 
filtration and by [DelSOl §1.6] is uniquely defined by the properties: 

(33) ad(e)(g>fc) C g>fc +2 and 

the map g>_fc/g>_fc+i -)■ g>A;/g>fc+i given by ad{e)^ is an isomorphism. 

It is easy to see that this filtration can be defined explicitly by 

(34) g>fc := ^ (Ker(ad(e)*) nlm(ad(e)*+^"^)) 

2 >max(l—fc,l) 

We will sometimes denote this also by g>fc-i or by Qe,>k- 

Let e"*" denote the orthogonal complement to {e} under the Killing form 
(•, •) and let 

(35) u := g>i, 0 := g>i, I := od(e)^(e-^) n D 

Lemma 5.1.1. (1) I is an ideal in u. 

(2) dimo/I = 1 and e ^ I. 

(3) There exists a unique sympleetie form u on u/o such that for any 
a,b £ u we have [a, b] —uj{d, b)e € I, where a and b denote the elasses 
of a and b. 

(4) u/I is a Heisenberg Lie algebra, and its center is spanned by class e 
ofe. 

Proof. Note that ad(e)^(g>_ 2 ) = g >2 = f and od(e)^(g>_ 2 ) = g> 2 - Pick 
an s[ 2 -triple (/, h, e). Using the h-grading it is easy to see that g>fc is a Lie 
algebra filtration. Now 
([I]) [u, /] C [u, o] = g >2 = od(e) 2 (g>_ 2 ) C I. 

([2|) Since e-*- fl g >_2 has codimension at most 1 in g>- 2 , I has codimen¬ 
sion at most 1 in 0 . Thus it is enough to show that e ^ I, i.e. if e = 
[e, [e, c]] then (e, c) / 0. Since ad{h) has integer eigenvalues, {h, h) / 0. 
Now [e, [e,-//2]] = [e,-h/2] = e and {e,-fl2) = {[e,-h/2],-f /2) = 
(— /i/2, [e, //2]) = — l/4(/i, h) 7 ^ 0. Now, c-|-//2 € ker(ad(e)^) C g>-i C e-*-, 
thus (e, c) = (e, -f/2) 7 ^ 0 . 

For dni) let c := (/, e)“^/ and define uj{d,b) := (c, [a,b]). It is easy to see 
that uj is the only anti-symmetric form satisfying [a, 6 ] — a;(a, b)e G I. Let us 
show that io is non-degenerate. Let a G u such that [a, b] £ I for any b £ u. 
This implies [c, a] G u-*- = g>_i and thus a G 0 . 

([^ follows immediately from ([3|). □ 

Definition 5.1.2. We now define a character of the center of u/I by re¬ 
quiring it to be 1 on e, consider the corresponding oscillator representation 
of the Lie group Exp(u/I) and lift it to an irreducible representation Se of 
U := Exp(u). We then define the generalized Whittaker model associated 
to e by Me ■= indfl(5e). 

The connection to the generalized Whittaker models W/ is given by the 
following straightforward lemma. 
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Lemma 5.1.3. Let {f,h,e) be an 5l2-triple and define ip £ g* by ip{x) := 
{f,e)~^{f,x). Then A4e is naturally isomorphic to W^p. 

Remark 5.1.4. The analogous approach in positive characteristic immedi¬ 
ately faces two problems: exponentials not being defined and the Killing 
form being degenerate. However, for 0 = we can replace the Killing 
form by the trace form, and try to replace the exponential by the map 
X I— 7 - Id-|-X. Then the next question is whether Lemma 15.1.11 holds. One 
can show that in three cases it fails completely: if charF = 2, n > 3, or 
charF = 3,n>8, orn> charF — 1 > 3. In these cases there exists e G 0 „ 
such that e G 0>3 and 1 = 0 . We also see in these cases that the Deligne 
filtration is not a Lie algebra filtration. 

In other cases we have, for any e G g, e G 0>2 but e ^ 0>3 . It is not 
clear whether u and 0 are always Lie subalgebras or whether I is an ideal in 
u. However, any e G 0 n can be completed to an s[ 2 -triple. Using this triple, 
one can show that if charF > 2 and = q then Lemma 15 .1.1 1 holds. 

Id-bu forms a subgroup of G which includes Id-|-/ as a normal subgroup, 
e defines a central character of the Heisenberg group (Id -|-u)/(Id +1) and 
one can consider the corresponding oscillator representation and Whittaker 
model. 

5.2. Degenerate Whittaker models. Let Z be a rational semi-simple 
element that commutes with e. For any t £ Q define 

(36) 0>f := X] ( 0>* ^ 0^ 

Note the following straightforward lemma. 

Lemma 5.2.1. If h £ gz is a neutral element for e then g'^f = 0>^^ 

In fact, commuting e and Z is the same amount of information as the Lie 
algebra element X = e + Z. We can reformulate the filtration in terms of 
X. First define 

(37) gj^* := ^ (Ker((ad(X) - t Id)*) n lm{{ad{X) - t Id)*+^)). 

2 >max(—Ai,0) 

The following lemma is straightforward. 

Lemma 5.2.2. 

X,t Z e j e,Z X.t 

g>fc = g* n g%^ and 9ft =2^ 2^ 0>) 

i t>s—i 

Now define 

(38) u := g>f 0 := g^f I := ad{X)‘^09 J := I + ad{X){u). 

Lemma 5.2.3. J is an ideal in u, and the algebra u/J is isomorphic to the 
Heisenberg algebra defined in 1)5.11 using the element e of the Lie algebra gz- 

Proof. Note that for any t 0, gf C X-^ and ad{X) is invertible on gf. 
Thus J n gf = u n gf. 
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To see that J is an ideal in u, let a G u H gf and b G J n gf. Then 
[a, 6] Gun gf+t, which lies in J unless s + t = 0. Ifs + t = 0 then 
[a,b] G (g^)> 2 , which lies in the ideal I' defined by e in (g'^)> 2 - 

We also see that u/J = (g^)> 2 //'. □ 

Definition 5.2.4. Using the isomorphism in Lemma 15.2.31 we define an os¬ 
cillator representation of the Lie group Exp(u//) and lift it to an irreducible 
representation Ue of U := Exp(u). We then define the degenerate Whittaker 
model associated to e and Z hy M.z,e '■= ind^(iTe). 

From the Lemmas 15.1.31 and 15.2.31 we obtain 

Corollary 5.2.5. Let Z £ q be a rational semi-simple element and let 
{f,h,e) G gz be an 5l2-triple. Let ip G g* be defined by pairing with {f,e)~^f. 
Then M.z,e is naturally isomorphic to yVh+z,if 

6. Global setting 

Let iL be a number field and let A = Ak be its ring of adeles. In this 
section we let y be a non-trivial unitary character of A, which is trivial on K. 
Then y defines an isomorphism between A and A via the map o i-a ya, where 
Xaib) = xi^b) for all 6 G A. This isomorphism restricts to an isomorphism 

(39) ^^{fi€A\fi\K^l} = {Xa\a£K}^K. 

Given an algebraic group G defined over K we will denote its Lie algebra 
by g and we will denote the group of its adelic (resp. AT-rational) points by 
G(A) (resp. G{K)). We will also define the Lie algebras g(A) and g{K) in 
a similar way. 

Given a Whittaker pair {S,ip) on g{K), we set u = g>;^ and n to be the 
radical of the form uj^\u, where oj^{X,Y) = ip{[X,Y]), as before. Let I C u 
be any choice of a maximal isotropic Lie algebra with respect to this form, 
and let U = exp u, N = exp n and L = exp 1. Observe that we can extend ip 
to a linear functional on g(A) by linearity and, furthermore, the character 
y^(expX) = y((/?(X)) defined on L(A) is automorphic, that is, it is trivial 
on L[K). We will denote its restriction to N{A) simply by Xip- 

Definition 6.0.1. Let (S', ip) be a Whittaker pair for g{K) and let U, L, N, Xp 
and y,^ be as above. For an automorphic function /, we define its {S,ip)- 
Whittaker-Fourier coefficient to be 

(40) WXsffif) := f Xpin)~^f{n)dn. 

Jn{a)/n{k) 

We also define its {S, ip, L)-Whittaker-Fourier coefficient to be 

( 41 ) ■■= [ x!^{i)-^fmi 

Jl(a)/l{k) 

Observe that and WFg ^ define linear functionals on the space of 

automorphic forms. If (vr, W) is an automorphic representation of G, then we 
will denote their restrictions to vr by and WFg^{ tt) respectively. 
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In order to adapt our arguments to the global setting we will have to 
replace Lemma 13.2.11 by the following one, which is analogous to |GRS11[ 
Propositions 7.2 and 7.3]. 

Lemma 6.0.2. Let (7r,I4-) be an automorphic representation of G. Then 
>VJ^5,<^(7r) 7 ^ 0 if and only ifWF^ ^{tt) / 0. More specifically, ifWiFs,ip{f) 7^ 
0 for some / G vr then WFg ^{'K{u)f) / 0 for some u G U{K). 

Proof. We assume that (p is non-zero since otherwise the statement is a 
tautology. Let / G vr be such that WTs^ipif) / 0. Define a function f^L on 
L by 

4r(0 = WJ-5,^(7r(/)/) 

and observe that the function • 4^ is left-invariant under the action 

of N{A)L{K). In other words, we can identify • 4^ with a function 

on 

(42) L{A)/N{A)L{K) - {L/N){A)/{L/N){K), 

where the equality follows from the fact that L/N is abelian. Therefore, we 
have a Fourier series expansion 

(43) 4^(0= c,p^^L{f)'if{l)x!^{l), 

ip<£{L{A)/N{A)L{K))'' 

where 

(44) c^,x4/)= [ m-^x!^{l)-^f{l)dl. 

^ Jl{a)/l{k) 

Since 

(45) 0^WT^Jf) = f^L{e)= 

^p<£(L{A)/N{A)L{K))'^ 

we conclude that at least one of the coefficients c^jj^^l (/) is different from 0. 

Now observe that the map X i—>■ uj^{X, ■) = ipoa,d{X) induces an isomor¬ 
phism between u/1 and (1/n)'. Hence, according to equations (l39|) and (I42|) . 
we can use the character y to define a group isomorphism 

{U/L){K) {L{A)/N{A)L{K)r 

U fju, 

where 

'f’uil) = x{t{[^i^]))^ u = exp X and l = expY. 

Hence, for all u G U{k) and I € L we have 

= xiTiix, Y]))x{p{Y)) = x(p(r + [^, 4 )) 

= X(7’(e''‘^^^^(^))) = x{T{Xd{u)Y)) = x^{ulu~^). 

Here we are taking again u = expX, I = expT and the middle equality 
follows from the vanishing of p on But now, from formula (1441) and the 






32 


RAUL GOMEZ, DMITRY GOUREVITCH, AND SIDDHARTHA SAHI 


fact that / is automorphic, we have 


^f{l)dl. = [ X^iulu ^f{l)dl. 

^ .Jl(a)/l(k) Jlc'' . 


IL{A)/L{K) 


Xw{l) V(w 'lu)dl. = WF^ {'K{u)f), 


-1 


Il{a)/l{k) 


for all u € U{k). Since we have already seen that at least one of these 
coefficients is nonzero, we obtain the result claimed in the lemma. □ 


The rest of the proof of Theorem can be applied in the adelic setting, 
with the appropriate modifications. This implies Theorem O 


Appendix A. Schwartz induction and the proof of Lemma 12.3.41 
We start with the following lemmas from functional analysis. 

Lemma A.0.1 i |Tre671 Theorem 50.1 and Proposition 50.1]). Let V and 
W be Hausdorff locally convex complete topological vector spaces. Suppose 
that V is a nuclear space. Then the projective and the injective topologies 
on V W agree, and we will denote the completion with respect to these 
topologies by V®W. Moreover, 

(i) V* is nuclear. 

(a) IfW is nuclear as well then Vis nuclear. 

(Hi) IfUcV is a closed subspace, then both U and V/U are nuclear. 

Here, V* and W* denote the strong dual. 

Lemma A.0.2 f |Tre67l formulas (50.18) and (50.19)]). Let V and W be 
Frechet spaces. Suppose that V is a nuclear space. Then 

(i) {V^W)* ^ WgW* 

(ii) L{V,W) ^ 

Here, L{V,W) denotes the space of all continuous linear maps form V to 
W, endowed with the compact-open topology. 

Lemma A.0.3 f |dC1911 Corollary 1.2.5 and Proposition 1.2.6]). 

(i) Let G he an affine real algebraic group. Then S{G) is a nuclear Frechet 
space and for any smooth Frechet representation n of G of moderate 
growth, we have a natural isomorphism 

S{G)m^S{G,7r). 

(ii) For two smooth affine semi-algebraic varieties M, N we have S{M)^S{N) = 
S{M X N). 

Lemma A.0.4. Given f € S{G;tt) and x, g €zG, we define 
{R{g)f){x) = f{xg) {R7r{g)f){x) = -K{g)fixg) 

{L{g)f){x) = f{g~^x) {L7r{g)f){x) = Tr{g)f{g-^x). 

Then, all of the induced G-module structures on S{G;Tr) are isomorphic. 

Proof. Given / G S{G]tt), let 

f{x) = f{x~^) for all X € G. 
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Then 

{R{9)f){x) = = f{x~^g) = f{g~^x) = {L{g)f){x). 

It’s clear, then, that the map / eA / dehnes a G-intertwining isomorphism 
between (i?,<S(G; vr)) and {L,S{G]tt)). Similarly, given / G <S(G;7r), we set 

f{x) = TT{x)f{x) for all x G G. 

Since r is of moderate growth, / G 5(G; tt) and 

R'^{g)f{x) = 7T{x){R7r{g)f){x) = TT{x)7r{g)f{xg) = f{xg) = {R{g)f){x), 

that is, the map / e->■ / defines a G-intertwining isomorphism between the 
spaces (i?7r, 5(G; tt)) and (i?, <S(G; vr)). The other isomorphisms are similar. 

□ 


Corollary A.0.5. Under any of the above G-module structures, 

5(G;7r)G = vr, 

where S{G]tt)g is the space of G-coinvariants, i.e. the quotient of S{G;t^) 
by the joint kernel of all G-invariant functionals. 

Lemma A.0.6. We have 

indg(p) = {S{G,p) (g> Ah)h- 

Proof. Let $ : 5(G, p) ind^(/9) denote the surjection that defines ind^(/9), 
see Definition 12.3.11 Observe that for all h ^ H, 

{^{Rp{h)f)){g) = I p{h){Rp{h)f{gh)dh = [ p{h)p{h)f{ghh)dh = Afl{h)^{f){g). 
■JH Jh 

Hence, the map defines an iJ-invariant operator between {Rp<^Ah,S{G] p)) 
and ind^(/3). Since this operator is surjective, it should factor through a sur¬ 
jective map 

(46) (5(G;/9) (g) A//)//^ indg(/9). 

We want to show that this map is injective. Let us fix T G [(5(G;/9) (g 
Ah)*]^• Now according to [dClQll Lemma 2.2.5], there exists a semi- 
algebraic open cover of G/H and a tempered partition of unity 

{7fc}fc=i subordinated to such that 

(47) p-\Uu) = Uk^H, 

where p : G —> G/H is the natural projection. Using the above equa¬ 
tion, we can define a partition of unity {rik}^=i subordinated to the open 
cover {p“^(Gfe)} consisting of Ll-invariant functions. Furthermore, given 
/g5(G;p), 

^{hkf) = lk^{f), k = l,...,n. 

Therefore, we have a decomposition 

n 

T = Y.VkT 

k=l 
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and we can identify each rj^T with an element of [{S{p ^(Uk)'iP) (8> ■ 

According to (H71) 

S{p-\Uk))=S{H)®S{Uk) 

and hence, according to Corollarv lA.0.5l [f5fp~^ff/t.l: p)i$^Au)*]^ = S{Uk]p)*, 
that is, for all A: = 1,..., n, there exists € S{Uk', p)* snch that 

PkT{f) = fkmpkf))=n{jkHf)) = {lkTkO^){f). 

But we can see as an element of ind^(/?)h Therefore, if we set T = 
Y^klkTk, then 

r = ^ PkT = ^ JkTk o $ = T o d>. 
k k 

We have thus shown that any element of [(5(G; p)(^ Ah)*]^ factors through 
ind^(/?)* which proves the injectivity of the map (H6l) . □ 

Proof of Lemma \2. We have 

HomG(indg(p),7r*) = (((S(G, p) (g> Aff)*)^§)7r*}^ = HomcxHiSiG, p}(g>Aff,7r*) 

HoniGxB(‘S(G,p)§7r 0 Aff,C) = HoinGxH(p§>‘S(G,7r) 0 Aff,C) 

Here, G acts on <S(G, vr) by Ltt, while H acts diagonally: on p (g> Ah and 
on S{G, it) by R. This action is isomorphic to an action in which G acts on 
iS(G, tt) by L and H by Rtt. Under this action we have 

HomGxLi(p§‘5(G, 7r)(g)AH,C) = ((<S*(G))‘^ig)7r*(g)p* (g) 

Since all left G-invariant distributions on G are proportional and right Ag- 
equivariant, the latter space is isomorphic to Hom/^(p,7r*Aj;j^^AG'). □ 
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